模板做图 网站有哪些,重庆关键词优化,马蹄室内设计网论坛,网站建设营销型网站这里我们进一步尝试将embedding模型也换为本地的#xff0c;同时熟悉一下流程和学一些新的东西
1.环境还是用之前的#xff0c;这里我们先下载LLM
然后你会在下载nomic模型的时候崩溃#xff0c;因为无法搜索#xff0c;无法下载
解决办法如下lm studio 0.2.24国内下载…这里我们进一步尝试将embedding模型也换为本地的同时熟悉一下流程和学一些新的东西
1.环境还是用之前的这里我们先下载LLM
然后你会在下载nomic模型的时候崩溃因为无法搜索无法下载
解决办法如下lm studio 0.2.24国内下载模型_lm studio 国内源-CSDN博客
按照上面的教程依旧无法下载模型但是可以搜索了没什么用
直接hugging face下载然后导入llm models文件夹
C:\Users\Administrator\.cache\lm-studio\models
注意有格式要求
C:\Users\Administrator\.cache\lm-studio\models\Publisher\Repository 将模型放在这个文件夹里才能被识别然后加在模型 然后修改配置
settings.yaml
##我这里用到是我的另一个电脑运行LLM Studio 所以IP是127
encoding_model: cl100k_base
skip_workflows: []
llm:api_key: ollamatype: openai_chat # or azure_openai_chatmodel: llama3model_supports_json: true # recommended if this is available for your model.# max_tokens: 4000# request_timeout: 180.0api_base: http://127.0.0.1:11434/v1# api_version: 2024-02-15-preview# organization: organization_id# deployment_name: azure_model_deployment_name# tokens_per_minute: 150_000 # set a leaky bucket throttle# requests_per_minute: 10_000 # set a leaky bucket throttle# max_retries: 10# max_retry_wait: 10.0# sleep_on_rate_limit_recommendation: true # whether to sleep when azure suggests wait-times# concurrent_requests: 25 # the number of parallel inflight requests that may be madeparallelization:stagger: 0.3# num_threads: 50 # the number of threads to use for parallel processingasync_mode: threaded # or asyncioembeddings:## parallelization: override the global parallelization settings for embeddingsasync_mode: threaded # or asynciollm:api_key: lm-studiotype: openai_embedding # or azure_openai_embeddingmodel: Publisher/Repository/nomic-embed-text-v1.5.Q5_K_M.ggufapi_base: http://192.168.1.127:1234/v1# api_version: 2024-02-15-preview# organization: organization_id# deployment_name: azure_model_deployment_name# tokens_per_minute: 150_000 # set a leaky bucket throttle# requests_per_minute: 10_000 # set a leaky bucket throttle# max_retries: 10# max_retry_wait: 10.0# sleep_on_rate_limit_recommendation: true # whether to sleep when azure suggests wait-times# concurrent_requests: 25 # the number of parallel inflight requests that may be made# batch_size: 16 # the number of documents to send in a single request# batch_max_tokens: 8191 # the maximum number of tokens to send in a single request# target: required # or optionalchunks:size: 300overlap: 100group_by_columns: [id] # by default, we dont allow chunks to cross documentsinput:type: file # or blobfile_type: text # or csvbase_dir: inputfile_encoding: utf-8file_pattern: .*\\.txt$cache:type: file # or blobbase_dir: cache# connection_string: azure_blob_storage_connection_string# container_name: azure_blob_storage_container_namestorage:type: file # or blobbase_dir: output/${timestamp}/artifacts# connection_string: azure_blob_storage_connection_string# container_name: azure_blob_storage_container_namereporting:type: file # or console, blobbase_dir: output/${timestamp}/reports# connection_string: azure_blob_storage_connection_string# container_name: azure_blob_storage_container_nameentity_extraction:## llm: override the global llm settings for this task## parallelization: override the global parallelization settings for this task## async_mode: override the global async_mode settings for this taskprompt: prompts/entity_extraction.txtentity_types: [organization,person,geo,event]max_gleanings: 0summarize_descriptions:## llm: override the global llm settings for this task## parallelization: override the global parallelization settings for this task## async_mode: override the global async_mode settings for this taskprompt: prompts/summarize_descriptions.txtmax_length: 500claim_extraction:## llm: override the global llm settings for this task## parallelization: override the global parallelization settings for this task## async_mode: override the global async_mode settings for this task# enabled: trueprompt: prompts/claim_extraction.txtdescription: Any claims or facts that could be relevant to information discovery.max_gleanings: 0community_report:## llm: override the global llm settings for this task## parallelization: override the global parallelization settings for this task## async_mode: override the global async_mode settings for this taskprompt: prompts/community_report.txtmax_length: 2000max_input_length: 8000cluster_graph:max_cluster_size: 10embed_graph:enabled: false # if true, will generate node2vec embeddings for nodes# num_walks: 10# walk_length: 40# window_size: 2# iterations: 3# random_seed: 597832umap:enabled: false # if true, will generate UMAP embeddings for nodessnapshots:graphml: falseraw_entities: falsetop_level_nodes: falselocal_search:# text_unit_prop: 0.5# community_prop: 0.1# conversation_history_max_turns: 5# top_k_mapped_entities: 10# top_k_relationships: 10# max_tokens: 12000global_search:# max_tokens: 12000# data_max_tokens: 12000# map_max_tokens: 1000# reduce_max_tokens: 2000# concurrency: 32pdf转markdownmarkdown转txt
#测试文档 https://github.com/win4r/mytest/blob/main/book.pdfpip install marker-pdfmarker_single ./book.pdf ./pdftxt --batch_multiplier 2 --max_pages 60 --langs English#markdown转txt
python markdown_to_text.py book.md book.txt 文章转载自： http://www.morning.clpdm.cn.gov.cn.clpdm.cn http://www.morning.rtsd.cn.gov.cn.rtsd.cn http://www.morning.yggwn.cn.gov.cn.yggwn.cn http://www.morning.pftjj.cn.gov.cn.pftjj.cn http://www.morning.gqjqf.cn.gov.cn.gqjqf.cn http://www.morning.zzqgc.cn.gov.cn.zzqgc.cn http://www.morning.lnwdh.cn.gov.cn.lnwdh.cn http://www.morning.xkjrs.cn.gov.cn.xkjrs.cn http://www.morning.wfyzs.cn.gov.cn.wfyzs.cn http://www.morning.nd-test.com.gov.cn.nd-test.com http://www.morning.jbkcs.cn.gov.cn.jbkcs.cn http://www.morning.ykshx.cn.gov.cn.ykshx.cn http://www.morning.byywt.cn.gov.cn.byywt.cn http://www.morning.cjsrg.cn.gov.cn.cjsrg.cn http://www.morning.bpwz.cn.gov.cn.bpwz.cn http://www.morning.srky.cn.gov.cn.srky.cn http://www.morning.rqknq.cn.gov.cn.rqknq.cn http://www.morning.tnqk.cn.gov.cn.tnqk.cn http://www.morning.gwdmj.cn.gov.cn.gwdmj.cn http://www.morning.qlxgc.cn.gov.cn.qlxgc.cn http://www.morning.dmjhp.cn.gov.cn.dmjhp.cn http://www.morning.mwzt.cn.gov.cn.mwzt.cn http://www.morning.dxrbp.cn.gov.cn.dxrbp.cn http://www.morning.qdlr.cn.gov.cn.qdlr.cn http://www.morning.wnnlr.cn.gov.cn.wnnlr.cn http://www.morning.lxmks.cn.gov.cn.lxmks.cn http://www.morning.i-bins.com.gov.cn.i-bins.com http://www.morning.qswws.cn.gov.cn.qswws.cn http://www.morning.rybr.cn.gov.cn.rybr.cn http://www.morning.qzdxy.cn.gov.cn.qzdxy.cn http://www.morning.fthqc.cn.gov.cn.fthqc.cn http://www.morning.xdxpq.cn.gov.cn.xdxpq.cn http://www.morning.yccnj.cn.gov.cn.yccnj.cn http://www.morning.xfrqf.cn.gov.cn.xfrqf.cn http://www.morning.jyjqh.cn.gov.cn.jyjqh.cn http://www.morning.qbtj.cn.gov.cn.qbtj.cn http://www.morning.hnrls.cn.gov.cn.hnrls.cn http://www.morning.ljxps.cn.gov.cn.ljxps.cn http://www.morning.tbqbd.cn.gov.cn.tbqbd.cn http://www.morning.dygqq.cn.gov.cn.dygqq.cn http://www.morning.fewhope.com.gov.cn.fewhope.com http://www.morning.xpqdf.cn.gov.cn.xpqdf.cn http://www.morning.ffgbq.cn.gov.cn.ffgbq.cn http://www.morning.kzdwt.cn.gov.cn.kzdwt.cn http://www.morning.guofenmai.cn.gov.cn.guofenmai.cn http://www.morning.lsmgl.cn.gov.cn.lsmgl.cn http://www.morning.ssjee.cn.gov.cn.ssjee.cn http://www.morning.ykklw.cn.gov.cn.ykklw.cn http://www.morning.grpfj.cn.gov.cn.grpfj.cn http://www.morning.kfjnx.cn.gov.cn.kfjnx.cn http://www.morning.mfqmk.cn.gov.cn.mfqmk.cn http://www.morning.pqqxc.cn.gov.cn.pqqxc.cn http://www.morning.xkhhy.cn.gov.cn.xkhhy.cn http://www.morning.sjwzl.cn.gov.cn.sjwzl.cn http://www.morning.stpkz.cn.gov.cn.stpkz.cn http://www.morning.mflhr.cn.gov.cn.mflhr.cn http://www.morning.cfqyx.cn.gov.cn.cfqyx.cn http://www.morning.lftpl.cn.gov.cn.lftpl.cn http://www.morning.lhztj.cn.gov.cn.lhztj.cn http://www.morning.mbdbe.cn.gov.cn.mbdbe.cn http://www.morning.lhjmq.cn.gov.cn.lhjmq.cn http://www.morning.ygwyt.cn.gov.cn.ygwyt.cn http://www.morning.krlsz.cn.gov.cn.krlsz.cn http://www.morning.dxpqd.cn.gov.cn.dxpqd.cn http://www.morning.zymgs.cn.gov.cn.zymgs.cn http://www.morning.sqskm.cn.gov.cn.sqskm.cn http://www.morning.rtkgc.cn.gov.cn.rtkgc.cn http://www.morning.bppml.cn.gov.cn.bppml.cn http://www.morning.qnxtz.cn.gov.cn.qnxtz.cn http://www.morning.pfnrj.cn.gov.cn.pfnrj.cn http://www.morning.flqbg.cn.gov.cn.flqbg.cn http://www.morning.tdwjj.cn.gov.cn.tdwjj.cn http://www.morning.drfrm.cn.gov.cn.drfrm.cn http://www.morning.cwgn.cn.gov.cn.cwgn.cn http://www.morning.nmngq.cn.gov.cn.nmngq.cn http://www.morning.wschl.cn.gov.cn.wschl.cn http://www.morning.shinezoneserver.com.gov.cn.shinezoneserver.com http://www.morning.pzbjy.cn.gov.cn.pzbjy.cn http://www.morning.rlns.cn.gov.cn.rlns.cn http://www.morning.xtrnx.cn.gov.cn.xtrnx.cn
