济南网站建设熊掌号,用dw怎么做网站后台,114网站制作,wordpress图片_转移oss问#xff1a;在st02中看到#xff0c;Program和Export/Import的Swap出现红的了#xff0c;这个是什么原因啊#xff0c;是不是对系统的性能有影响啊#xff0c;是否应该调整一些参数啊。要怎么调整呢#xff1f; 复1#xff1a;双击红色的部分就可以看到相应的参数修改…
问在st02中看到Program和Export/Import的Swap出现红的了这个是什么原因啊是不是对系统的性能有影响啊是否应该调整一些参数啊。要怎么调整呢 复1双击红色的部分就可以看到相应的参数修改提示啦不过曾经有人告诉过我只要Hitratio的数字是99点多Swap是红色也没有什么关系我也不知 道这样的说法是不是正确的请各位高手指正
复2不是很正确现在看起来Export/import的Buffer设置是偏小了因为看起来系统启动了6个小时Export/import的SWAP总数就超过1W了远高于SAP推荐的警戒值1000/day 建议观察一段时间如果的确是平均SWAP超过1000/day的话还是调高一下Export/import buffer比较好。
复3跟你说这话的人还是有点片面我不敢苟同这种观点。 我刚做的一个项目客户情况比较夸张buffer参数完全没有调整保持着安装完成的初始状态整个系统SWAP情况极其严重特别是Program buffer的SWAP平均超过100,000/day但你看 Hitratior的话可能觉得还好也有97.8%。实际上呢系统的Free物理内存还有超过600M这种情况完全属于资源分配不合理实际运用中他们在执行一些比较大的程序是速度特别慢因 为执行的过程中在拼命的SWAP本来内存里能处理的东西被搞到SWAP里折腾能不慢吗 我认为只要有SWAP你都应进行评估要尽可能发挥系统最大的潜力。有一家公司的系统我特佩服31H的老系统了用了n多年了我刚做SAP的时候他们生产系统就已经运行了有1、2年 了业务量也是巨大的家用电器生产厂PP、SD、FI、MM都在用并发用户也是很多的可是performance也是不错的你看人家的SWAP几乎总是为0我是一次也没看到过在他们生产系统有SWAP赞 再罗嗦一句Hitratior低也不能代表系统的Buffer没配好比如说系统刚系统的一段时间里你会看到Hitratior可能很低这个很正常以Program buffer为例程序系统第一次启动后总归是不在Buffer里的运行一次后才会在Buffer里这也就是为什么系统刚启动的时候SAP用起来好像比平时要慢对Buffer的评价最好是长期观察、分析这样的结果更有说服力。
复4SWAP的单位一律为 “次”。 to:Aming你们的Export/Import确实小了10个小时下来都超过7W了看看物理内存还空吧如果空可以调一调看。
复5hehe 这是德式数字你大概用SAP*看的换个用户把用户的default里面的数字表现方式改一下。另外关于SWAP的问题要分开来看如果你只有一台应用服务器例如CIDB或者CIDB尤其是你们上了多个模块后各种Buffer会很快的上涨ABAP尤其如此而且你会觉得是个无底洞这是如果要追求好的performance需要增加DI。另外建议你看看SAP library里关于Memory management的文档里面描述SAP内存管理的模式懂了模式才可以掌握如何调整它。 回复: Good discuss but I still would like to ask Mr. Performance answer this kind of question. Not because I do not know he need making money from this kind of task. I do not recommend Jr. basis doing any performance tuning task before you understand whole system technical architechture.
You should start from following oss note if you have no idea about Performance optimums via memory mangement. Its really depends on the OS and DB and R3 release. -------------------------------------------------------------------- OSS NOTE: 103747
Performance in 4.0/4.5/4.6: Parameter recommendations:
Performance in 4.0/4.5/4.6: Parameter recommendations Help SAP Note Number Note Language DEENJA Display Versions 4950515253 Print SSCR Download
Text Header Data Release Correction Instructions Support Packages Symptom You want to set R/3 Memory Management. Only an optimum setting of the R/3 Memory Management ensures error-free operation of the R/3 System and satisfactory system performance.
After upgrading to R/3 Release 4.0, you notice long response times in the R/3 System.
Last changes:
03/11/1999: New recommendations for abap/heap_area_dia 11/23/1998: Release for customer (ts) 11/13/1998: Additions for AS/400 (rh) 11/17/1998: Revision prior to release for customers (ts) 11/18/1998: AS/400: Implementation of Note 126210 (rh) 01/28/2000: Note 192658: Parameters for SAP BW 02/02/2000: abap/heap_area... and Shared Memory Limit on SINIX 09/03/2000: Released for 4.6A/B 08/28/2000: AS/400: Including recommendations for Rel. 4.6 (rh) 10/26/2000: AS/400: New recommendations for em/initial_size_MB (4.5B) 03/06/2001: Shared Memory Management for S/390 Application Server 05/17/2001: Note on em/global_area_MB (- Note 329021) 11/07/2001: Remove AS/400-specific notes (rh) 02/07/2002: Linux 2.4: Note 386605 for MM
Additional key words Memory Management, EarlyWatch, GoingLive, Basis course BC315 Workload Analysis, Release 4.0 Cause and preconditions This note is a collection of individual notes on Memory Management for Release 4.0. The recommendations are intended as initial settings (that is, generally) before or immediately after a release upgrade. After production start individual parameters may have to be corrected and the actual requirements adapted. This note is NOT valid for the IBM iSeries (AS/400) platform. The additional features of this platform are described in other notes (see item 7 Additional notes). In Note 192658 you can find additional recommendations for BW Systems. Solution *********************************************************************** * Support by TCC Services * ***********************************************************************
In order to ensure good performance after the upgrade, we recommend that you make use of the TCC services, for example within the Early Watch service contract or within the Going Live functional upgrade check.
You have to plan for two service sessions, approx. 4-8 weeks before and 2-8 weeks after the upgrade. The following is required to ensure a good performance:
Before the upgrade: Plausibility check for the hardware sizing Recommendations on the configuration after the upgrade After the upgrade: Detailed adjustment of the R/3 and database parameters Identification of unsolved performance problems
Make the dates for the TCC service session in time before the upgrade, in order to avoid long waiting times for the date !!! 0) Whats new in Release 4.0 ? a) Recommendations for the physical memory: ----------------------------------------
Note 89305 contains information 89305 Resource requirements for Release 4.0A/4.0B 128221 Increased memory consumption with Oracle 8 85524 R/3 scaling (Quicksizer)
The following recommendation for settings apply to application servers with: 750 MByte main memory 50 users 3 R/3 modules being used
For development and test systems and for very small productive systems smaller settings may also be possible. b) Zero Administration Memory Management for Windows NT: ----------------------------------------------------- In Release 4.0A, the Zero Administration Memory Management described in R/3 Note 88416 is active for R/3 under Windows NT. We recommend that you proceed according to this note for NT. The parameters affected by the Zero Administration Memory Management are marked with (*) in this note. All recommendations in this note for parameters with (*) therefore only apply to UNIX platforms (except for LINUX, see below). c) Standard profiles for UNIX / 4.0B: ---------------------------------- With new 4.0B installations, memory profiles are delivered which depend on the physical main storage facilities. The parameter RAM which the person performing the installation makes available for the R/3 instance is used as input. This results in an area according to the following table:
Area │ RAM [MB] ----------------------------------------------------------------- 0 │ up to 750 - no recommendations given by this note! I │ 750 RAM 1024 II │ 1024 RAM 2048 III │ 2048 RAM - not for HP-UX and SINIX
Remark: Owing to operating system restrictions, area III cannot be set for SNI/SINIX and HP-UX with 32-bit kernel.
Parameters are now set depending on the respective area. The parameters which are set due to the new installation procedure correspond to a large extent to the recommendations of this note.
With 4.0A installations or upgrades to 4.0, the parameters should be changed according to this note either before or immediately after the upgrade.
Note 146528 provides extensive information regarding the limitations and options regarding the configuration of machines with a lot of RAM ( 2GByte). d) Export/import buffer / ATP server: ---------------------------------- An ATP service can be started on an R/3 instance of an R/3 System. This should be the instance on which the enqueue service is running. In the Default.pfl rdisp/atp_server ATP instance (for example, rdisp/atp_server us0311_ALR_21) must be set. If the ATP service is active, Transaction SM51 displays the value Atp in the Service type column. The ATP service stores its data in the export/import buffer. This buffer on the ATP instance should therefore be configured larger than usual (20 MBytes). (Other programs also use the export/import buffer.) See also Note 99999. e) AIX: ---- For large AIX systems ( 1 GByte physical main memory), refer to Notes 146528, 98084, 95260, 95454, 117267, 124555, and 128935. CAUTION: Pay attention to the warning in Note 95260 or 117267 !!! a) Digital Unix: ------------- Refer to Notes 120539 and 25960.
b) HP-UX and SNI/SINIX (32-bit) ---------------------------- With HP-UX, the allocatable shared memory is limited to approx. 1700 MB for each server; for SNI/SINIX to approx. 1700 MB for each instance. Therefore, no settings must be made according to area III for these operating systems. If em/initial_size_MB and rdisp/ROLL_SHM (in area II) are set greater than the default values 750 (MB) or 8000 (64 MB), other shared memory elements (for example, buffers) must be decreased.
Solution for SINIX: Set up several R/3 instances on one computer or use the 64-bit R/3 kernel. Solution for HP-UX: Several R/3 instances are not useful since the SHM limitation applies per computer. Therefore, use the 64-bit R/3 kernel.
For the 64-bit kernel also refer to Notes 146528 and 146289.
You can search profiles for errors at the OS level using test program sappfpar nameSID nr (or sappfpar pffull path of profile (see Note 31395).
c) Zero Adminstration Memory Management for Linux: ------------------------------------------------------
With Release 4.5, a simplified memory management has been implemented for R/3 on Linux 2.4 as well. The following parameters are set automatically: ztta/roll_area, ztta/roll_first, ztta/roll_extension, rdisp/ROLL_MAXFS, rdisp/PG_MAXFS, rdisp/ROLL_SHM, rdisp/PG_SHM, em/initial_size_MB. You can also keep parameters abap/heap_area_dia, abap/heap_area_nondia, abap/heap_area_total on the default values defined by the SAP System for the time being. We recommend you to proceed as described in Note 386605. d) Miscellaneous: -------------- The changes for the parameters rsdb/ntab/entrycount and rsdb/ntab/ftabsize are particularly large due to the namespace extension (that is, table name lengths from 10 - 40 characters). The sizes have more than doubled. Parameters which are new or are changed significantly as compared to 3.1 are indicated with !!!. 1. R/3 buffer
Area-independent: rsdb/ntab/entrycount 30000 !!! rsdb/ntab/irbdsize 4000 rsdb/ntab/sntabsize 2500 rsdb/cua/buffersize 5000 sap/bufdir_entries 10000 zcsa/table_buffer_area 50000000 zcsa/db_max_buftab 10000 rtbb/buffer_length 30000 rtbb/max_tables 500 Area-dependent: I II III rsdb/ntab/ftabsize 20000 30000 30000 !!! zcsa/presentation_buffer_area 15000000 20000000 20000000 abap/buffersize 240000 240000 400000 Remark on program buffer (abap/buffersize):
A program buffer with 240 MByte is in most cases not sufficient for Release 4.0/4.5. During production operation it should be enhanced, if required. For this, observe the swaps in the program buffer in transaction ST02. Some 1,000 swaps a day can be tolerated, but if you find more than 10,000 swaps a day, the buffer should be enhanced. Note that imports of ABAP programs cause buffer fragmentation and consequently swaps. To avoid this, reduce the number of transports in your production system! The enhancement of program buffers (abap/buffersize) can cause problems in the memory administration: Windows NT: PXA_NO_SHARED_MEMORY because of address space fragmentation. Defective basis addresses of NT system DLLs fragment the address space of work processes. The large related Shared Memory Segments (PXA, Extended Memory, ...) possibly cannot be created, because the available free address space is being fragmented by these DLLs. This leads to a start with the NOT-PXA buffer of 4 MB. As a workaround the parameter em/address_space_MB can be reduced to 250 MByte (also refer to the internal Note 129813). AIX: Note 98084 AIX: ABAP programm buffer PXA larger than 256MB HP-UX and SNI: Because of the Shared Memory restrictions (see above), an enhancement of the program buffer can lead to error PXA_NO_SHARED_MEMORY. In this case other Shared Memory Segments have to be made smaller. With HP-UX you have to consider the following: If the total out of program buffer and EM is higher than 1000 MB, the program buffer has to be in a pool: if abap/buffersize em/initial_size_MB 1000 MB. ipc/shm_psize_06 -10 and ipc/shm_psize_02 -10 (This measure is necessary to avoid a Shared Memory fragmentation.) The buffer sizes specified are standard installation values. After the production startup, you have to adjust the parameters to the actual requirements. To do this, call Transaction ST02. If displacements occur in column Swaps you should increase the buffers in question (however, note that up to 10,000 swaps may occur in the program buffer). Export/import buffer:
The settings recommendations for the export/import buffer are area-independent and depend on the application programs you use. As standard settings we recommend: Release 4.0/4.5: rsdb/obj/buffersize 4096 (KB) !!! rsdb/obj/max_objects 5000 (buffer entries) !!! rsdb/obj/large_object_size 8192 (bytes) !!! As of Release 4.6: rsdb/obj/buffersize 20000 (KB) !!! rsdb/obj/max_objects 20000 !!! rsdb/obj/large_object_size 8192 (bytes) !!!
If the export/import buffer is swapped a lot (more than 1000 swaps/day) then it should be enlarged. 2. Local memory for work processes (roll memory)
*********************************************************************** ******** Parameters with (*) only apply to UNIX. *********** ******** For NT see Note 88416. *********** ******** For Unix see Note 386605. *********** ***********************************************************************
ztta/roll_area (*) 6500000
ztta/roll_first (*) 1 3. Roll buffer and extended memory (except Windows NT)
Area-independent: ztta/roll_extension(*) 500000000 !!! AIX: 2000000000
(Note: When using the alternative Memory Management on AIX (see Notes 95454 and 124555), ztta/roll_extension must be set as is recommended in these notes.) rdisp/ROLL_MAXFS (*) optimal: 32000, minimal: 16000 rdisp/PG_MAXFS (*) optimal: 32000, minimal: 16000 rdisp/ROLL_SHM(*) 8000 (up to 16000) !!! rdisp/PG_SHM(*) 1000 em/global_area_MB for SAP kernel 4.6D: 24 (default) for SAP kernel 4.6D: 10% of em/initial_size_MB, maximum (see also Note 329021) 255 - em/blocksize_KB / 1024 (AIX) 10% of em/address_space_MB (Windows) 255 (others) Area-dependent: I II III em/initial_size_MB(*) 512 750 (to 1000) 1000 (to 2000) The specified values em/initial_size_MB and rdisp/ROLL_SHM are the standard installation values. After production startup, you must adjust the parameters to the actual requirements. To do this, call Transaction ST02. If the line Extended Memory contains In memory - max. use 20% of In memory
you should enlarge the em/initial_size_MB (within the scope of the values entered in parentheses above).
In other words: 20% extended memory should also be free at times of high load.
If the line Roll area contains Max. use In memory you should increase rdisp/ROLL_SHM (within the above-mentioned values specified in parentheses). 4. Heap memory (except Windows NT)
abap/heap_area_dia and abap/heap_area_nondia (new as of 03/11/1999): In general, we recommend setting abap/heap_area_dia(*) abap/heap_area_nondia(*) as follows: AIX:
Standard: 180000000 (180 MByte) If ES/TABLE SHM_SEGS is used (Notes 95454 124555) 80000000 ( 80 MByte) HP-UX:
up to HP-UX Releases smaller than 10.20: 600000000 with OS parameter: maxdsize 640 MByte as of HP-UX Release 10.20, the following size is possible: 920000000 with OS parameter: maxdsize 944 MByte
SNI (SINIX): 350000000 Note: Value 350000000 for abap/heap_area(non)dia is a standard value which can be used for almost all SINIX/Reliant systems. In rare cases however, memory management errors can occur. In this case you may be able to solve the problem by reducing the parameter further (for example, 250000000). DEC: 900000000
Sun: Standard: 900000000 (900 MByte)
Caution: If you set em/initial_size_MB 1500, you must reduce
abap/heap_area_dia and abap/heap_area_nondia (see also
section 8f) in Note 97497). Windows: Delete the profile paramter from the profile abap/heaplimit(*) 20000000 Area-dependent: I II III
abap/heap_area_total (*) 750000000 750000000 1500000000 5. Basic information on the parameter settings You must save your old profiles before the change. If the instance with the new parameters does not start, you can use the old profile. All parameter settings must be checked before being used (program sappfpar, in the test system and so on). Consider the architecture restrictions of the operating systems (for example, Shared Memory restrictions). If you do not consider these restrictions errors and terminations may be the result. Consider that actual recommendations on the parameter settings by the TCC can only be provided within the service sessions.
***********************************************************************
WARNING: If the R/3 memory areas are set up too small (R/3 buffers, R/3 Extended Memory, and so on), unnecessary performance problems may occur. If the memory areas are set up too large, errors can occur when you start the R/3 instance, or runtime errors may occur. The optimum settings are dependent on the hadware (CPU and RAM), the R/3 release and the operating system release. The TCC Services GoingLive and EarlyWatch will provide you with individual recommendations for your system.
*********************************************************************** 6. Miscellaneous
Also see the sections Swap Space, File System Cache, Dependencies of the operating systems, and Questions and answers in Note 97497. 7. Additional notes: Notes for IBM AS/400 139326 AS/400: Memory Management in Release as of 4.6A 126210 AS/400: nametab buffer larger than 16MB 123418 AS/400: Performance 4.0B on AS/400 121625 AS/400: buffer size 119754 AS/400: Parameter for nametab buffer 110157 AS/400: Generic table buffer 79478 AS/400: ERROR ShmCreate (...) 73628 AS/400: MCH3601 in rstsac2/convert_ABAP_to_C/9 44695 Memory Management in Release as of 3.0C, AS/400
Notes on IBM S/390 326949 Shared Memory Management S/390 Application Server
8. Questions and answers
(See also Questions and answers in Notes 33576 and 44695 and the SAP TechNet article Memory management as of Release 3.0C (http://sapnet.sap-ag.de/TechNet - System Monitoring - Knowledge Base - New Memory Management.)
For background knowledge on memory management of R/3 you can also refer to the book R/3 Performance Optimization (ISBN 0-7821-2563-8) a) Why is em/initial_size_MB in area III (RAM 2000 MByte) set to 2000 MByte in so restrictive a manner?
The main reason is that the 32-bit architecture available to us at present can manage an address space of a maximum of 2GB - 3.8GB (depending on the operating system). For category III, the address space is made up as follows: em 1024MB (shared) PXA 400MB (shared) Roll 64MB (shared) sonst. 150MB (shared) heap 1500MB (local) stack 50MB (local) .......... 3138MB b) Which parameters should be changed for prevention ?
Parameters ztta/roll_extension, rdisp/roll_MAXFS, abap/heap_area_(non)dia, ABAP/heap_area_total, and rdisp/pg_MAXFS do not influence the performance (in the sense of velocity). They are only stability-relevant, that is, they prevent programs from terminating. These parameters should therefore be changed for prevention. Example: Even if the system never needs 256 MB of paging (rdisp/pg_MAXFS 32000), you should not economize on the paging file. It is senseless to wait until the year-end settlement terminates since the paging area was configured too small and only afterwards is the paging area to be configured on request. Similar examples apply to the other parameters.
As opposed to this, the buffer sizes (abap/buffer_size ... ) only influence the response time. In the case of emergency you can wait until the buffer starts to swap and then raise the response time. (However, you can also set it large enough immediately).
Source code corrections
回复6 to: Ben newcat 好久没来, 谢谢两位还记得我 to:amingschwalbe 你们的系统export/import的参数设置的确是有些问题. 但并不是你export/import的buffer大小不够,而是你系统中为export/import预留的最大object数值不够大. 系统当前的object数为2000个且已经被用完了,所以尽管还有82.07%和22.91%的export/import空间却不能够被利用. rsdb/obj/max_objects 2000 改到 5000 rsdb/obj/buffersize 4096 改到 8192 ( for schwalbe) 情况会有改善的. schwalbe 你的program buffer size 有点小了,我不知道你的软硬件具体情况所以不好给出具体的数字. 如果是32位R/3跑在unix系统下的话尤其要小心,仔细调整pool 10和40的值,不然系统很可能起不来. 我个人认为没有必要一定要刻意追求所有的SWAP一直是0. 回复7多谢LARRY我仔细看才发现了问题所在德文的格式中“”和“。”刚好是调转的
相关
Whats the min swap system? Depends on release, kernel(32/64) and platforms(NT/UNIX/LINUX/AS400)... Basis idea for MIN SWAP for 20-30current users: 3 X Memory(i.e. 1GB memory needs at least 3GB memory. But on the Basis Certification TEST the answer is 3.5GB(3 X Memory 500MB) For 64bit kernel, please use at least 18-20GB. Use the whole disk if its possible for beter performance James Yen - 颜心德
来在disk array里留两个备用磁盘还是有好处的。。。 swap太小会出Memory问题导致DUMP的。 SAP官方Swap Space Requirements、SAP Memory Management 其它讨论
首先确认SAP服务器多久没有重启了如果是有几个月没有重启则ST02的SWAP一般会出现红色这个一般重启SAP可以release swap的问题。
如果SAP重启几天后SWAP就出现红色根据SAP服务器的硬件可以适当调整RZ10的参数。
1、调整前一定先把/usr/sap/DEV/SYS/Profile进行备份这个非常重要。
2、对具体哪个参数进行调整T-code: ST02在出现红色的地方“双击”进入另一个界面然后点菜单栏的“当前参数”就能看到它对应的Profile Parameter Value Unit Comment然后根据它的现有值 x 0.1 要调整的数值。
T_Code:RZ10(调整)RZ11查看参数值、默认值和参数说明
eg: swap出现红色双击后对应参数值rtbb/buffer_length 10000 kB 那么在RZ10把这个参数调整为11000
3、RZ10调整后要进行激活激活后要重新启动SAP MMC如果出现调整后SAP MMC无法启动那么重启SAP服务器在重启SAP MMC如果还是无法起动那么把刚备份的原Profile目录覆盖新的Profile目录然后再重新调整根据它的现有值 x 0.05 要调整的数值。
调整后再观测ST02的变化如果原来都没有调整过那一般要按照上面的调整方式调整几次ST02就相对比较正常了 文章转载自： http://www.morning.dljujia.com.gov.cn.dljujia.com http://www.morning.tdqhs.cn.gov.cn.tdqhs.cn http://www.morning.qbksx.cn.gov.cn.qbksx.cn http://www.morning.xjpnq.cn.gov.cn.xjpnq.cn http://www.morning.bnrnb.cn.gov.cn.bnrnb.cn http://www.morning.ghslr.cn.gov.cn.ghslr.cn http://www.morning.nktgj.cn.gov.cn.nktgj.cn http://www.morning.routalr.cn.gov.cn.routalr.cn http://www.morning.tgcw.cn.gov.cn.tgcw.cn http://www.morning.zxcny.cn.gov.cn.zxcny.cn http://www.morning.lfttb.cn.gov.cn.lfttb.cn http://www.morning.lfpdc.cn.gov.cn.lfpdc.cn http://www.morning.beeice.com.gov.cn.beeice.com http://www.morning.pnfwd.cn.gov.cn.pnfwd.cn http://www.morning.ylyzk.cn.gov.cn.ylyzk.cn http://www.morning.slnz.cn.gov.cn.slnz.cn http://www.morning.jyjqh.cn.gov.cn.jyjqh.cn http://www.morning.lzwfg.cn.gov.cn.lzwfg.cn http://www.morning.ndpzm.cn.gov.cn.ndpzm.cn http://www.morning.srbmc.cn.gov.cn.srbmc.cn http://www.morning.bgxgq.cn.gov.cn.bgxgq.cn http://www.morning.qrnbs.cn.gov.cn.qrnbs.cn http://www.morning.tnjz.cn.gov.cn.tnjz.cn http://www.morning.rxnl.cn.gov.cn.rxnl.cn http://www.morning.jwfkk.cn.gov.cn.jwfkk.cn http://www.morning.rxlck.cn.gov.cn.rxlck.cn http://www.morning.qrmyd.cn.gov.cn.qrmyd.cn http://www.morning.wkhfg.cn.gov.cn.wkhfg.cn http://www.morning.nktxr.cn.gov.cn.nktxr.cn http://www.morning.tynqy.cn.gov.cn.tynqy.cn http://www.morning.jlmrx.cn.gov.cn.jlmrx.cn http://www.morning.ljygq.cn.gov.cn.ljygq.cn http://www.morning.jsmyw.cn.gov.cn.jsmyw.cn http://www.morning.rzjfn.cn.gov.cn.rzjfn.cn http://www.morning.lpnb.cn.gov.cn.lpnb.cn http://www.morning.qxmnf.cn.gov.cn.qxmnf.cn http://www.morning.rqgbd.cn.gov.cn.rqgbd.cn http://www.morning.gjssk.cn.gov.cn.gjssk.cn http://www.morning.cbczs.cn.gov.cn.cbczs.cn http://www.morning.rqhdt.cn.gov.cn.rqhdt.cn http://www.morning.fbmzm.cn.gov.cn.fbmzm.cn http://www.morning.qxycf.cn.gov.cn.qxycf.cn http://www.morning.crqbt.cn.gov.cn.crqbt.cn http://www.morning.lwlnw.cn.gov.cn.lwlnw.cn http://www.morning.mgnrc.cn.gov.cn.mgnrc.cn http://www.morning.yxshp.cn.gov.cn.yxshp.cn http://www.morning.plqsc.cn.gov.cn.plqsc.cn http://www.morning.kcwkt.cn.gov.cn.kcwkt.cn http://www.morning.fgrkc.cn.gov.cn.fgrkc.cn http://www.morning.jpgfx.cn.gov.cn.jpgfx.cn http://www.morning.xpfwr.cn.gov.cn.xpfwr.cn http://www.morning.ktsth.cn.gov.cn.ktsth.cn http://www.morning.jjnry.cn.gov.cn.jjnry.cn http://www.morning.ntyks.cn.gov.cn.ntyks.cn http://www.morning.ndfwh.cn.gov.cn.ndfwh.cn http://www.morning.rhph.cn.gov.cn.rhph.cn http://www.morning.mgtrc.cn.gov.cn.mgtrc.cn http://www.morning.ytbr.cn.gov.cn.ytbr.cn http://www.morning.tldfp.cn.gov.cn.tldfp.cn http://www.morning.rlbc.cn.gov.cn.rlbc.cn http://www.morning.xckqs.cn.gov.cn.xckqs.cn http://www.morning.jcxqc.cn.gov.cn.jcxqc.cn http://www.morning.zrrgx.cn.gov.cn.zrrgx.cn http://www.morning.xppj.cn.gov.cn.xppj.cn http://www.morning.rlwcs.cn.gov.cn.rlwcs.cn http://www.morning.c7501.cn.gov.cn.c7501.cn http://www.morning.bgpb.cn.gov.cn.bgpb.cn http://www.morning.pmdzd.cn.gov.cn.pmdzd.cn http://www.morning.dtrz.cn.gov.cn.dtrz.cn http://www.morning.gcqs.cn.gov.cn.gcqs.cn http://www.morning.wxccm.cn.gov.cn.wxccm.cn http://www.morning.crxdn.cn.gov.cn.crxdn.cn http://www.morning.duckgpt.cn.gov.cn.duckgpt.cn http://www.morning.dnvhfh.cn.gov.cn.dnvhfh.cn http://www.morning.wfjyn.cn.gov.cn.wfjyn.cn http://www.morning.bpwdc.cn.gov.cn.bpwdc.cn http://www.morning.nbgfz.cn.gov.cn.nbgfz.cn http://www.morning.pqqhl.cn.gov.cn.pqqhl.cn http://www.morning.sxcwc.cn.gov.cn.sxcwc.cn http://www.morning.hwycs.cn.gov.cn.hwycs.cn
